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Abstract 

In a sample of 6.6 million produced B mesons we have observed decays 
B -> rfK, with branching fractions B{B + -► rfK+) = (6.5±^;| ± 0.9) x 1(T 5 
and B{B° -> r/if ) = (4.7tl;o ± 0.9) x 10 ~ 5 - We have searched with com- 
parable sensitivity for 17 related decays to final states containing an r/ or r/' 
meson accompanied by a single particle or low-lying resonance. Our upper 
limits for these constrain theoretical interpretations of the B — * rj'K signal. 
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The dominant decay modes of B mesons involve the b — > c quark transition with coupling 
to a W + boson. For many of these modes the decay amplitude may be described by a tree 
diagram in which the light quark (spectator) is bound in both the initial B meson and 
final charmed hadron via soft gluon exchange. With recent improvements in experimental 
sensitivity, less favored modes are becoming accessible. These include: h — > u tree diagram 
transitions that are suppressed by the small Cabibbo-Kobayashi-Maskawa [jlj (CKM) matrix 
element V u b, such as B — > ixtv p|; effective flavor changing neutral current (FCNC) decays 
b — ► s described by loop diagrams, such as the "electromagnetic penguin" B — > K*j 0; 
and decays to charmless hadrons such as B — ► Ktc |§-f|. The hadronic decays may be 
classified according to contributions to the amplitude from the several tree and penguin 
diagrams shown in Fig. [I] |7|f|. Some of these charmless hadronic decays offer prospects 
for the observation of CP violation, while others facilitate the quantitative understanding 
of the amplitudes that are essential to the interpretation of future CP measurements. For 
example, the decays B — > t]K and B — > rj'K, with B — > Kir, have been examined in this 
context MM. 
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(c) (d) 

FIG. 1. Feynman diagrams describing the representative decays B + — > rj^K^*^: (a, b) internal 
penguins; (c) external tree; (d) flavor-singlet penguin. 



In this paper we present results of experimental searches for B meson decays to two-body 
final states containing 77 and rj' mesons. These 1 = mesons are mixtures of flavor-SU(3) 
octet and singlet states, the latter being of particular interest because of its allowed formation 
through a pure (two or more) gluon intermediate state (Fig. [I] (d)). 

The data were accumulated at the Cornell Electron-positron Storage Ring (CESR). The 
integrated luminosity was 3.11 fb _1 for the reaction e + e~ — > T(4S') — > BB (center-of-mass 
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energy E cm = 10.58 GeV). This luminosity corresponds to the production of 3.3 x 10 6 charged 
and an approximately equal number of neutral B mesons. In addition we recorded 1.61 fbT 1 
of data with E cm below the threshold for BB production to measure continuum processes. 

The CLEO II detector [I I ] emphasizes precision charged particle tracking, with specific 
ionization (dE/dx) measurement, and high resolution electromagnetic calorimetry based on 
CsI(Tl). From the raw data we reconstruct charged pions and kaons, photons (from 7r°, 
77, and t]' decays), and 7r + 7r~ pairs that intersect at a vertex displaced by at least 3 mm 
from the collision point ("vees", from — > 7r + 7r~). Candidate B decay tracks must meet 
specifications on the number of drift chamber measurements, goodness of fit, and consistency 
with an origin at the primary or particular secondary vertex. Candidate photons must be 
isolated calorimeter clusters with a photon-like spatial distribution and energy deposition 
exceeding 30 MeV. We exclude photon pairs from extremely asymmetric 7r° or 77 decays to 
reject soft photon backgrounds, requiring |cos 6 1 * | < 0.97, where 9* is the meson center of 
mass decay angle relative to its flight direction. We reject charged tracks and photon pairs 
having momentum less than 100 MeV/c. 

We fit photon pairs and vees kinematically to the appropriate combined mass hypothesis 
to obtain the meson momentum vectors. Resolutions on the reconstructed masses prior to 
the constraint are about 5 — 10 MeV/c 2 (momentum dependent) for 7r° — > 77, 12 MeV/c 2 
for rj — > 77, and 3 MeV/c 2 for K® — > 7r + 7r _ . Information about expected signal distributions 
with the detector response comes from a detailed GEANT based simulation of the CLEO 
detector [12| that reproduces the resolutions and efficiencies of data in a variety of benchmark 
processes. 

Since the B mesons are formed nearly at rest, while the B daughters we observe are 
relatively light, the latter have momenta close to half of the beam energy (2.6 GeV/c). 
For this reason the final states are well separated from those involving heavier daughters, 
i.e., the dominant b — > c decays. The principal signatures for the selected decay modes 
are consistency of the resonance decay invariant masses with the known masses and widths 
of those resonances, and kinematic consistency of the total final state with the B meson 
(mass and energy). Because the beam energy Eb is better known than the reconstructed 



B meson energy Eb, we substitute the former in the B mass calculation: M = ^ El — p 2 B , 
with ps the reconstructed B momentum. We define also the variable AE = Eb — Ei,. The 
measurement resolution on M is about 2.6 MeV/c 2 , and on AE it is 25-40 MeV, depending 
on the apportionment of the energy among charged tracks and photons for each mode. 

For vector-pseudoscalar decays of the B and pj decays of the rf we gain further discrim- 
ination from the helicity variable 7i (cosine of the vector meson's rest frame two-body decay 
angle relative to its flight direction), which reflects the spin alignment in the decay. For 
modes in which one daughter is a single charged track, or is a resonance pairing a charged 
track with a tt°, we achieve statistical discrimination between kaons and pions by dE/dx. 
With Sk and S n defined as the deviations from nominal energy loss for the indicated par- 
ticle hypotheses measured in standard deviations, the separation Sk — S n is about 1.7 at 
2.6 GeV/c. 

The main backgrounds arise from continuum quark production e + e~ — > qq. We discrim- 
inate against these jet-like events with several measures of the energy flow pattern. One is 
the angle 9bb between the thrust axis (axis of maximum energy projection magnitude) of the 
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candidate B and that of the rest of the event. For a fake B candidate selected from particles 
belonging to a qq event those particles tend to align with the rest of the event, whereas the 
true B decays have a thrust axis that is largely uncorrelated with the tracks and showers 
from the decay of the partner B. We reject events with |cos#bb| > 0.9. In addition we use 
a multivariate discriminant T incorporating the energy deposition in nine cones concentric 
with the event thrust axis, and the angles of the thrust axis and ps with respect to the e + e~ 
beam direction ||. We have checked the backgrounds from the favored B decay modes by 
simulation and found their contributions to the modes in this study to be negligible. 

To extract event yields we perform unbinned maximum likelihood fits to the data, in- 
cluding sidebands about the expected mass and energy peaks, of a superposition of expected 
signal and background distributions: 

N 



C{N S , N B ) = e -( Ns+N ^ J] [NsV s 0; + N B V B (f, 



Here V s and V b are the probability distribution functions (PDFs) for signal and continuum 
background, respectively. They are functions of observables X; for event i, and of parameters 
(3 and 7 (discussed below). The form of £ reflects the underlying Poisson statistics obeyed 
by Ns and Nb, the (positive-definite) numbers of signal and continuum background events, 
respectively, whose expectation values sum to the total number N of input events. Observ- 
ables for each event include M, AE, T y and (where applicable) resonance masses and 7i. 
Where two modes involve a charged hadron (generically h + ) that is either ir + or K + we fit 
both simultaneously, with C expanded so that the signal and background yields of both n + 
and K + are fit variables. In this case the PDFs depend also on the dE/dx observables S n 
and Sk- The number of events N for these fits ranges from ~ 30 to a few thousand. 

The PDFs V s an d Vb are constructed as products of functions of the observables Xj. The 
dependences of V s on masses and energies are Gaussian, double Gaussian, or Breit-Wigner 
functions, whose means, widths, etc. appear as the parameters (3 in Eq. [1]. The background 
PDF Vb contains signal-like peaking components in its resonance mass projections, to ac- 
count for real resonances in the background, added to smooth components for combinatoric 
continuum. The smooth components are low-order polynomials, except that for M we use 



an empirical shape [13] that accounts for the phase space limit at M = E^. The dependences 
of both V s and Vb on JF, Sk, and SV are bifurcated Gaussian functions. We obtain the 
parameters (3 of V s from separate fits to simulated signal, and 7 of Vb from fits to data in 
a sideband region of the AE — M plane. 



Results for our 42 B decay chains [nj] appear in Table |. The row label subscripts 
denote secondary decays, including rf — > ?77r + 7r~ with i] — > 77 (rjmr), 7/ — > ?77r + 7r~ with 
i] — ► 7T + 7r~7r° (57r), and r] — > 7r + 7r~7r° (3tt). The table gives each branching fraction quoted 
as central value with statistical followed by systematic error, or as 90% confidence level 
upper limit. We include systematic errors from uncertainties in the PDFs, i.e., in j3 and 7, 
obtained from a Monte Carlo convolution of the likelihood function with Gaussian resolution 
functions for these parameters, including their most important correlations. This procedure 
changes the upper limit by less than 10% in most cases. We also include systematic errors 
for reconstruction efficiencies and selection requirements, and quote upper limits computed 
with efficiencies one standard deviation below nominal. 
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TABLE I. Measurement results. Columns list the final states (with secondary decay modes 
as subscripts), event yield from the fit, reconstruction efficiency e, total efficiency with secondary 
branching fractions B s , and the resulting B decay branching fraction B. 
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TABLE II. Combined branching fraction results, with expectations from theoretical models. 



Decay mode 
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Where we have measured a given B decay mode in more than one secondary decay 
channel we combine the samples by adding the \ 2 = ~ 2 In £ functions of branching fraction 
and extracting a value with errors or limit from the combined distribution. The limit is the 
value of B below which 90% of the integral of £ lies. The results are summarized in Table 
m together with previously published theoretical calculations P,|HS|-|iT||. 

We have analyzed each of the decays also without use of the likelihood fit, employing 
more restrictive cuts in each of the variables to isolate the signals. The results are consistent 
with those quoted in the tables, but with larger errors (less restrictive limits) in most cases. 
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We find positive signals in both charge states of B — > rj'K: B(B + — ► rj'K + ) = 
(6.5±i;| ± 0.9) x 10" 5 and B(B° -»■ r]'K°) = (4.7l|;S ± 0.9) x 10" 5 . (The first error quoted 
is statistical, the second systematic.) The significance, defined as the number of standard 
deviations corresponding to the probability for a fluctuation from zero to our observed yield, 
is 7.5 for B + — > i]'K + and 3.8 for B° — > rj'K . The likelihood functions from the fits for 
B — > r]'h + and B° — > ^'X are shown in Fig. 0. For these modes we show also in Fig. [3] the 
projections of event distributions onto the M axis. Clear peaks at the B meson mass are 
evident. 

The observed branching fractions for B — > r^'-ft', in combination with the upper limits for 
the other modes in Table |TT] and with recent measurements of B — > i^7T and B ^ ttti ||, 
provide important constraints on the theoretical picture for these charmless hadronic decays. 
A large ratio of B — > r/iT toB -> 77.fr, consistent with our measurements, was predicted ]TB| 
in terms of interference of the two penguin diagrams in Fig. |I](a) and (b), constructive for 
B — > r/iT and destructive for B —> rjK. The effective Hamiltonian calculations M contain 

1E,20|1, light quark masses [2C], the QCD scale, and the effective 



uncertainties in form factors 
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FIG. 3. Projections onto the variable M. Overlaid on each plot as smooth curves are the best 
fit functions (solid) and background components (dashed), calculated with the variables not shown 
restricted to the neighborhood of expected signal. The histograms show (a) B + — > r]'h + with 
rj — > r/TTTT (r/ —* 3tt, dark shaded), rj' — > rjirn (rj — > 77, light shaded), and r( — > pj (open); (b) 



n 



K° with rj' — > ?77r7r (shaded) and rj' — > (open). 
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number of colors. They generally employ spectator and factorization |2T|] approximations. 
The unexpectedly large branching fraction for B — > rj'K has led to a reevaluation of some 
of the older calculations. Recent suggestions include contributions from the QCD gluon 
anomaly or other flavor singlet processes (Fig. [I] (d)) in constructive interference with the 
penguins [TU],^2]-|2I| . Prospects are good for resolution of some of these issues as new data 
become available. 
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